Repository navigation
e2e: add multi-node rollout test on EKS - #240
Open
ptalgulk01 wants to merge 2 commits into
Open
ptalgulk01 wants to merge 2 commits into
ptalgulk01 wants to merge 2 commits into
Conversation
ptalgulk01
force-pushed
the
scale-test-multi-node
branch
from
October 19, 2026 13:25
1c34f6a to
1d6aa4e
Compare
alicefr
reviewed
Oct 21, 2026
| // cannot afford enough memory/CPU for multi-node scale testing. | ||
| func TestMultiNodeRollout(t *testing.T) { | ||
| e2eutil.Providers(t, "bink", "eks") | ||
| t.Parallel() |
Collaborator
There was a problem hiding this comment.
Maybe it will be better to run this serially for the resource contention in bink
|
|
||
| // Verify the daemon pod actually reaches Running on every node before | ||
| // waiting for Idle - a pod stuck in e.g. ImagePullBackOff would | ||
| // otherwise just show up as a confusing Idle-wait timeout below. |
Collaborator
There was a problem hiding this comment.
This is a good check, probably we could generalize in a function and add the check to the other tests as well, in this way we fail fast. But not for this PR
Collaborator
Author
There was a problem hiding this comment.
Agreed — I'll generalize the daemon-pod Running check into a helper and wire it into the other tests in a follow-up PR, so failures surface fast everywhere.
Collaborator
|
It looks good :) |
Collaborator
|
@ptalgulk01 I think you need to increase the timeout for the overall tests |
Signed-off-by: Prachiti Talgulkar <ptalgulk01@users.noreply.github.com>
ptalgulk01
force-pushed
the
scale-test-multi-node
branch
from
October 21, 2026 12:45
1d6aa4e to
3c15272
Compare
Collaborator
|
@ptalgulk01 there is a missing signed-off and also please use the tag |
TestMultiNodeRollout takes ~24 min on the bink runners, pushing the full suite past the previous 60m budget (k8s 1.35 died with panic: test timed out after 1h0m0s while two tests were still in flight). On EKS the suite now provisions five EC2 nodes across three tests, which does not reliably fit the old 30m either. e2e: 60m -> 90m, e2e-eks: 30m -> 60m. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Prachiti Talgulkar <ptalgulk01@users.noreply.github.com>
ptalgulk01
force-pushed
the
scale-test-multi-node
branch
from
October 22, 2026 13:19
3c15272 to
75b6808
Compare
alicefr
approved these changes
Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage so far only exercises a single node per pool, so staged rollout across a fleet has never been tested. TestMidRolloutImageChange does provision two nodes, but it runs on bink only.
TestMultiNodeRollout provisions three nodes in one pool, rolls the pool to a new image, and asserts every node boots the new digest and the pool reports all nodes updated. It declares both bink and eks; the GitHub runners cannot host three nodes, so in practice this provides coverage on EKS.
maxUnavailable is asserted inside the same polling loop that waits for convergence, so the invariant is sampled throughout the rollout. Asserting it once up front would pass trivially, as UpdatingCount is zero before the rollout starts. A violation aborts immediately instead of retrying, which would let a later poll mask it.
A node that stays Degraded past a grace period aborts the wait and reports the daemon's own error. Degraded is cleared on the next successful reconcile, so a retried transient error must not fail the test; without the early abort, a failure such as an unauthorized registry pull surfaced only as an opaque matcher timeout minutes later.
Each node's daemon pod is also checked for Running before waiting for Idle, so a pod stuck in ImagePullBackOff is named directly.